Degenerate conformally invariant fully nonlinear 

elliptic equations 



YanYan Li* 
Department of Mathematics 
Rutgers University 
110 Frelinghuysen Road 
Piscataway, NJ 08854 
USA 



1 Introduction 

There has been much work on conformally invariant fully nonlinear elliptic equations and 
applications to geometry and topology See for instance [17], [5], [4], [10], [14], [9], and 
the references therein. An important issue in the study of such equations is to classify 
entire solutions which arise from rescaling blowing up solutions. Liouville type theorems 
for general conformally invariant fully nonlinear second order elliptic equations have been 
obtained in [14]. For previous works on the subject, see [14] for a description. Classifi- 
cation of entire solutions to degenerate equations is also of importance, as demonstrated 
in [6]. In this paper we give Liouville type theorems for general degenerate conformally 
invariant fully nonlinear second order elliptic equations. 

Let S nxn denote the set of n x n real symmetric matrices, S+ Xn denote the subset of 
gnxn COIls i s ting of positive definite matrices, 0(n) denote the set of n x n real orthogonal 
matrices, [/ C 5 fixn be an open set, and F E C X (U) fl C°(U). 

We list below a number of properties of (F,U). Subsets of these properties will be 
used in various lemmas, propositions and theorems: 

Q- 1 UO = U, VOGO(n), (1) 
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M <E U and N G S+ Xn implies M + N e U, (2) 
M <E U implies aM G £7 for all positive constant a, (3) 

{al | a > 0} n dU = 0, (4) 

where I denotes the n x n identity matrix. 
Let F G C\U) n C°(T7) satisfy 

F(0~ 1 MO) = F(M), V M <E U,y O <E 0(n), (5) 

(i^(M)))>0, VMeC/, (6) 
where F^M) := ^-(M), and, 

F > in 17, F = on dU, (7) 

Examples of such (F, U) include those given by the elementary symmetric functions. 
For 1 < k < n, let 

l<il<---<ij;<n 

be the A;— th elementary symmetric function and let be the connected component of 
{A G R n | o- fe (A) > 0} containing the positive cone T n := {A = (Ai, • • • , A„) | A; > 0}. Let 

U k := {M G S nxn | A(M) G T fc }, 

and 

F fc (M) :=<7 fc (A(M))*, 

where A(M) denotes the eigenvalues of M . Then (F, U) = (F fc , £4) satisfy all the above 
listed properties, see for instance [3]. 

Other, much more general, examples are as follows. Let 

r C K n be an open convex symmetric cone with vertex at the origin 

satisfying 

r n crcr 1 :={AGR n |^A l >o}. 

i 

Naturally, Y being symmetric means (Ai, A 2 , • • • , A n ) G V implies (A^, X i2 , ■ • ■ , X in ) G T for 
any permutation (i l5 i 2 , ■ ■ ■ , i n ) of (1, 2, • • • , n). 
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Let 

/ e C\T) nC°(T) 

satisfy 

f\ar = 0, VfE T n on I\ 

and 

f(s\) = sf(X), V s > and A G T. 

With such (/,r), let 

U := {M G S nxn | A(M) GT}, 

and 

F(M) := /(A(M)). 

Then (F, [/) satisfies all the above listed properties. In fact, for all these (F, U), A u E U 
implies Au < — see below for the definition of A u . So for these (F, U), the assumption 
Au < in various theorems in this paper is automatically satisfied. We note that in all 
these examples, F is actually concave in U, but this property is not needed for results in 
this paper. 

As mentioned above, entire solutions to general equation 

F(A U ) = 1, u > 0, A u G U, in R n 
are classified in [14]. Here and throughout the paper we use notation 

ah 2 n +2 9 2n 2n 2 2n „ 

n-2 (n-2) 2 (n-2) 2 1 1 

where V« denotes the gradient of u and V 2 it denotes the Hessian of u. 
In this paper we classify appropriate weak solutions to 

F(A U ) = 0, u > 0, in K n . 

The techniques developed in [14] play important roles in our studies. As in [14], we make 
use of the method of moving spheres, a variant of the method of moving planes which fully 
exploits the conformal invariance of the problem. The method of moving planes has been 
used in classical works of Gidas, Ni and Nirenberg [7] and Caffarelli, Gidas and Spruck 
[2], and others — see for instance [14] for a description, to study Liouville type theorems. 

For x G R n , A > 0, and for some function u, we denote the Kelvin transformation of 
u with respect to B\{x) by 
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Here and throughout the paper we use B a (x) C M n to denote the ball of radius a and 
centered at x, and use B a to denote B a (0). Also, unless otherwise stated, the dimension 
n is bigger than 2. 

We first introduce a notion of weak solutions to the degenerate equations. 

Definition 1.1 Let U C S nxn be an open set and F G C\U) n C°(U) satisfy (1), (2), 
(3), (5), (6), and (7). A positive continuous function u on an open set Q ofR n is said to 
be a weak solution of 

F(A U ) =0 into (8) 
if there exist {u,j} in C 2 (Vt) and {fii} in C°(£l,S nxn ) such that 

^ > 0, A Ui + Pi G U inVl, (9) 

and, for any compact subset K of Q, 

Ui^u, &->0, inC°(K), (10) 

F{A u '+f3i)^0 inC°(K). (11) 



In R n , n > 2, we use A to denote the set of functions u with the following properties: 
(Al) u G C^R"), u > in K™, and A-u < in R n in the distribution sense. 
(A2) There exists some 5 > such that for all < 5 < 5, all x G R™, all A > 0, and all 
bounded open set f2 of {y G R n | |y — x| > A}, 

(1 + 5)m > Mj;^ in and (1 + 5)u > u Xt \ on dQ imply (1 + 5)u > u Xy \ in Vt. 

Theorem 1.1 For any u G A, there exist x G R n and constants a > and b > snc/i 



-u ( x ) = ( i n r 



n~2 
2 



ViGl". (12) 



Remark 1.1 7/6 = 0, £/ien u = u(0). 

Remark 1.2 Theorem 1.1 can be viewed as an extension of the classical result which 
asserts that positive harmonic functions in M n are constants. 
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Theorem 1.2 Let U C S nxn be an open set and F G (^(U) n C°(U) satisfy (1), (2), 
(3), (4), (5), (6), and (7). Assume that a positive function u G C 1 (M n ) is a weak solution 
of 

F{A U ) = in R n , 

and satisfies 

Au < in M n in the distribution sense. (13) 

Then u = u(0) in R n . 

Remark 1.3 Our notion of weak solutions includes those arising from rescaling blowing 
up solutions. 

Theorem 1.3 Let U C S nxn be a convex open set and F G C 1 ^) H C°(U) satisfy (1), 
(2), (3), (4), (5), (6), and (7). Assume that a positive function u G C 1 ' 1 (R n ) satisfies 
(13) and 

F(A u )=0, or equivalently A u G dU, almost everywhere in R n . (14) 
Then u = u(0) in R n . 

Remark 1.4 IfU C {M G S nxn \ Trace(M) > 0}, then a weak solution or C 1 ' 1 solution 
u of F(A U ) = automatically satisfies Au < 0. In particular, this is the case for (F, U) = 
(Fk, Uk) for all k in all dimensions n. 

Remark 1.5 It was proved by Chang, Gursky and Yang in [6] that positive C 1 ' 1 (R 4 ) 
solutions to F2(A U ) = are constants. Aobing Li proved in [11] that positive C 1 ' 1 (R 3 ) 
solutions to F2(A U ) = are constants, and, for all k and n, positive C 3 (R n ) solutions to 
F k (A u ) = are constants. Our proof is completely different. 

We give in the following a notion of weak solutions to more general equations. 

Definition 1.2 Let U C S nxn be an open set, F G C l (U) r\C°(U), and h be a continuous 
function on an open subset Q ofR n . A positive function u G C°(Q) is said to be a weak 
solution of 

F{A u )<h mn (15) 

if there exist {ui} in C 2 (f2) and $ in C°(Q,S nxn ) such that, for any compact subset K 
of VI, (9), (10) hold, and 

[F{A U > + Pi) - h} + -> in C°{K), (16) 



6 



where we have used the notation w + := max{w,0}. 
Similarly we define that u is a weak solution of 

F{A U ) >h inn (17) 

by changing [F(A U > + fc) - h} + to [h - F(A U > + #)]+ in (16). 
We say that u is a weak solution of 

F(A U ) = h mVL 

if it is a weak solution of both (15) and (17). 

We also establish the following results concerning degenerate equations on R n \ {0}. 

Theorem 1.4 Let U C S nxn be an open set and F e C\U) n C°(U) satisfy (1), (2), 
(3), (4), (5), (6), and (7). Assume that a positive function u G C 1 (R n \ {0}) is a weak 
solution of 

F(A u ) = mR n \{0}, 

and satisfies 

Au < in R n \ {0} in the distribution sense. (18) 

Then 

u x ,x(y) <u(y), V 1 < A < \x\,\y- x\ > X,y ^ 0. (19) 
Consequently, u is radially symmetric about the origin. 

Theorem 1.5 In addition to the hypotheses on (F,U) in Theorem 1.4, we assume that 
U is convex. Assume that a positive C 1,1 (]R n \ {0}) function u satisfies (18) and 

F (A u ) = 0, or equivalently A u e dU, almost everywhere in R n \ {0}. 

Then (19) holds and, consequently, u is radially symmetric about the origin. 

Our proofs of Theorem 1.4 and Theorem 1.5 make use of a result in a companion 
paper [15]. 

Let tp be a C 1 function near 1 satisfying ip{l) = 1, and let, for a function v, and for 
x,y E M n , A close to 1, 

x, A; y) := <p(\)v(x + Ay). 
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Theorem 1.6 ([15]) Let Q C M. n fre a bounded open set containing the origin 0. We 
assume that u G C°(fl \ {0}) 7 v is C l in some open neighborhood ofVL, 

v > in H, (20) 

An<0 ra(l\{0}, (21) 
u>ui7ifi\{0}. (22) 

Assume that tp is as above and 

<p'(l)v(y) + Vv(y)-y>0, Vi/Gfi, (23) 

and assume that there exists some e 4 > such that for any \x\ < e 4 and |A — 1| < e 4 , 

inf [u — &(v, x, A; •)] = implies lim inf [u(y) — $(t>, x, A; y)] = 0. (24) 
n\{o} ' |»|-»o 



T/ien either 



lim inf — v(x)] > 0, (25) 

|xH0 



or u = v = v(0) near the origin. 

The paper is organized as follows. In Section 2, we prove Theorem 1.1. In Section 
3, we give some properties of weak solutions and C 1 ' 1 solutions. In particular, we give 
comparison principles, see Propostion 3.1 and Proposition 3.2, for weak solutions. A 
crucial ingredient in our proof of the comparison principles is Lemma 3.7, "the first 
variation" of the operator A u . Theorem 1.2 and Theorem 1.3 are proved in Section 3 
by first showing that u belongs to A and then showing that the b in (12) must be zero. 
In Section 4 we first establish some further comparison principles, Proposition 4.1 and 
Proposition 4.2, which allow the presence of isolated singularities. Then we prove Theorem 
1.4 and Theorem 1.5. Proofs of results in this section hevily rely on Theorem 1.6, a result 
in the companion paper [15]. 

2 Proof of Theorem 1.1 

Proof of Theorem 1.1. By (Al) and the maximum principle, 

lim inf \y\ n ~ 2 u(y) > 0. 

\y\-HX> 
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As in the proof of lemma 2.1 in [16] or [10], for any x in R n , there exists Xq{x) > such 
that 

u x ,\(y) < u(y), V < A < A (x), \y - x\ > A. 
For any 5 G (0, 5), we define 

X s (x) := snp{fi > | u X:X (y) < (1 + S)u(y), V < A < n, \y - x\ > A}. 

If X$(x) < oo for some x, then 

u xMx) (y) < (1 + V\y-x\> h(x). (26) 

Lemma 2.1 IfXs(x) < oo /or some < 5 < 5 and x G R n , then 



liminf|y| n (l + 8)u(y)-u xMx) (y) 



0. 



Proof. Suppose the contrary then for some < S < S, x G R n , \s(x) < oo, and for some 
R > 1 + Xs(x) and ei > 0, we have 



(1 + 5)u(y) - u x>h (y)\ > 2e u V \y - x\ > R. 
As in the proof of (27) in [16], there exists e 2 > such that 



e 2 



(1 + 5)u(y) - u X:X (y) > I^pr^' V |A - X s (x)\ < e 2 , \y - x\ > R. 



Since 



(27) 
(28) 



(1 + 5)u{y) - u xMx) (y) = 5u{y) > 0, V \y - x\ = X 5 (x), 
there exists < e 3 < e 2 such that 

(1 + 5)u{y) > u XtX (y), V |A - X s (x)\ < e 3 , \y - x\ = X. 

Let 

Q := {y G R n \ X s (x) < \y - x\ < R}. 
We know from (26), (27) and (28) that 

(1 + 5)u > u xMx) in Q, (1 + S)u > u xMx) on dVL. 

Thus, by (A2), 

(1 + S)u > u xMx) on n. (29) 

With (27) and (29), the moving plane procedure can go beyond X$(x), violating the 
definition of A^x). Lemma 2.1 is established. 

□ 
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Lemma 2.2 For all < 5 < 5 and for all x e R n , 



X s (x) n - 2 u(x) = (1 + 5) liminf \y\ n ~ 2 u(y). (30) 

|j/|-»oo 



Proof. Let Xs(x) < oo for some < 5 < 5 and some x £ R n . By Lemma 2.1, 

(1 + 5) liminf |y|»- 2 «(y) = lim lyr't^wM = A 5 (x)"- 2 «(x) < oo. (31) 

For any xGI", 

(1 + 5)u(y) > u x , x (y), V < A < A 5 (x), |y - x\ > A. 
Multiplying the above by \y\ n ~ 2 and sending \y\ to infinity leads to 

{1 + 5) liminf \y\ n ' 2 u{y) > X n - 2 u{x), V < A < \ s (x). (32) 

|j/|->oo 

We deduce from (32) and (31) that 

\ 5 {x) n ~ 2 u(x) < X s (x)u(x) < oo, V x e R n . 

Switching the roles of x and x leads to (30) in the case that Xs is not identically equal 
to infinity. On the other hand, if A«5 = oo on K n , we send A to oo in (32) to obtain (30). 
Lemma 2.2 is established. 

□ 

If A,5 = oo on R n for all < 5 < 5, then 

(1 + 8)u(y) > u XjX (y), V < 5 < 5, x G R n , \y - x\ > X > 0. 

Sending 5 to in the above yields 

u(y)>u x , x (y), VxGK", \y-x\ > A > 0. 

This implies u = u(0), see for example lemma 11.2 in [16]. 

We only need to consider the case that for some < 5 < 5, Xs is not identically equal 
to infinity. According to Lemma 2.2 and Lemma 2.1, Xs( x ) < 00 f° r & h J 

(1 + 8)u(y) > u xMx) (y), V x e K", \y - x\ > X s (x), 
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and 



liminf \y\ n (1 + 5)u(y) - u xMx) (y) =0, ViG 



For xGI", let 



A«5(a:) 2 (?/-a;) 



|y - x\ 



, i/>(y) ■-- 



w 



(x) 



U,(x) = U,(x) , , 



where we have used the notation := \J^\ 2» (u o ip) with being the Jacobian of ip. 
It is not difficult to see that is C 1 near 0, 

«,(*)(()) = A 5 (x) n " 2 M(a;) = (1 + 5) liminf \y\ n ~ 2 u(y) = liminf (1 + <J)tty(?/) > 0, 

|j/|->oo |s/| — ^0 

< (1 + <S)tty in Bs(x) \ {0} for some S(x) > 0. 
Following the proof of theorem 1.3 in [14] (see also [12] and [13]), we obtain 

Vw {x) {0) = W 0) (0), ViGl™ 

and then 



a 



2 

n-2 



n-2 
2 



d + |y - x\ 



V)/Gl n , 



where x G R n , d > and 



a 5 = (1 + 5) liminf |?/| n 2 u(y) > 0. 

Ij/Hoo 



Theorem 1.1 is established. 



3 Properties of weak solutions and the proof of The- 
orem 1.2 and Theorem 1.3 

We start with some properties of weak solutions. 

Lemma 3.1 Let U C S nxn be an open set satisfying (2). Assume that for a positive C 2 
function u in some open subset Q ofR n , there exist {ui} C C 2 (Q) and {/%} C C°(Q, S nxn ) 
such that (9) and (10) hold on any compact subset K of CI. Then 



A u eU in ft. 



(33) 
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Proof. For any i 6 fl, we fix some 5 > such that B 2 s(x) C f2. Consider, for small 
e > 0, 

u e {y) := - -\y - x\ 2 , y E B s {x). 
We know from (10) that 

Ui(x) = u e (x) + o(l), 
Ui (y) = u e (y) + ^eS 2 + o(l), y E dB s (x), 

u t (y)>u e (y) + o(l), yEB 5 (x), 

where o(l) — > as % — > oo, uniform in y and e. 

It is easy to see that for some a\ — 1 + o(l), yf E B s (x), 

Ui > a\u e on Bs(x), (34) 

«i(^)=a t V(y?). (35) 
Passing to a subsequence in (35), yf — > y e , -u(y e ) = w(2/ e ) — f |?/ e — x| 2 . So 

lim y\ = x. 

By (34) and (35), 

A^{yf)<A a >\yl). (36) 

Thus, by (9) and (2), A< u \yl) + $(yf) E 17. Sending i to oo, we have, using (10), 
A u \x) E U. Sending e to 0, we have A u (x) E U. Lemma 3.1 is established. 

□ 

Lemma 3.2 Let U C <S nxn be an open set satisfy (2) and F E C l (U) fl C°(U) satisfy 
(6). Assume that for a positive C 2 function u in some open set Q ofR n , there exist 
{u,i} C C 2 (tt) and {A} C C°(Q,<S nxn ) such that (9) and (10) hold for any compact 
subset K of CI, and, for some h E C°(Q), 

[ h _ F (A«i + _ o, mClM- (37) 

Then (33) holds and u is a classical solution of (17). 

Proof. We know from Lemma 3.1 that (33) holds. Following the proof of Lemma 3.1 
from the beginning until (36). Then, by (9), (2), (36) and (6), 
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and 

F (A Ul (y-) + < F (A a >\y-) + . (38) 

Since 

lim +&(»?)) =A«'(x), 

we have A" 1 *' (x) G £/ and, using the continuity of F on U, 

lim F (A*"'(ri) + m)) = F (A u \x)) . 

Sending % to oo in (38) leads to, in view of (37), 

f{A u \x)) > h(x). 

Sending e to 0, we obtain 

F(A u (x)) > h(x). 

Lemma 3.2 is established. 

□ 

Lemma 3.3 Let U C S nxn be an open set satisfy (2), F G C\U) n C°(U) satisfy (6). 
Assume that for a positive C 2 function u in some open set Vt of R n , there exist {ui} C 
C 2 (tt) and {fa} C C°(tt,S nxn ) such that (9), (10) and 

sup sup |V 2 -Uj| < oo (39) 

i K 

hold for any compact subset K of ft, and, for some h G C°(Q), 

[F(^+/%)-ft] + -0, mCfjO). (40) 
Then (33) holds and u is a classical solution of (15). 

Proof. We know from Lemma 3.1 that (33) holds. For any x G Q, we fix some 5 > such 
that B 2 s(x) C Q. Consider, for small e > 0, 

u £ (y) :=u{y) + | \y - x\ 2 , y G B s (x). 

Arguing as in the proof of Lemma 3.1, we find a\ = 1 + o(l), y\ — > x, 

A^(yl)>A a >\yl). (41) 
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Clearly, there exist a e > and 7 « > satisfying a e — > as e — > and 7 j — >■ as z — > oo, 
such that 

^"'(tf) + Piivt) > A u (x) - aj - 7i I. (42) 
We already know that A u (x) G U. So, by (41), (42), (2) and (6), 

A*(yf)+p i (yt)+a e + 'y i IeU 

and 

F (A u (x)) < F {A^{yD + + aj + 7i 7) . 

Because of (39), (10) and the positivity and the continuity of u, A Ui (yl)+f3i(y^)+a e I + %I 
remain bounded. Thus, by the continuity of F, 

F (A u (x)) < F (A u *( y *) + A(% £ )) + o(l) + o e (l), 

where o e (l) — > as e — > 0, uniform in z, and o(l) — > as z — > oo, uniform in e. Sending z 
to oo and then e to 0, we obtain, using (40), 

F{A u {x)) <h{x). (43) 

Lemma 3.3 is established. 

□ 

Lemma 3.4 In Lemma 3.3, we drop assumption (39) but add A u e U in Q. Then u is a 
classical solution of (15). 

Proof. Follow the proof of Lemma 3.3 until (42). Since we know that A u (x) e U, the 
right hand side of (42) is also in U for large z and small e. Thus 

F (A u (x) - aj - % I) < F {A^{yf) + ^{yf)) . 

Sending z to oo and then e to 0, we obtain (43). Lemma 3.4 is established. 

□ 

Lemma 3.5 Let Q C K n be an open set. Assume that a positive function u G C°(Q) is 
a weak solution of (8). Then, for any constant b > and for any x G IR n ; the function 
v(y) := b^~u(x + by) is a weak solution of 

F(A v ) = zn tl := {y G R n \ x + by G SI}. 
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Proof. It is obvious. 



Lemma 3.6 Let Q C M. n fre an open sei Assume that a positive function u G C°(Q) is a 
weak solution of (8). Then, for any x G M n and X > 0, u x> \ is a weak solution of 



F(A U * 







in n 



x,\ 



y e 



X 



A 2 (?/ - x) 

\y — x \ 2 



e n 



Proof. This follows from the conformal invariance of the operator F(A U ), see for example 
line 9 on page 1431 of [10]. 

□ 

The following is a comparison principle for weak solutions. 

Proposition 3.1 Let U C S nxn be an open set satisfying (2) and (3),F G C l {U)r\C G (JJ) 
satisfy (6) and (7), and let Q C R n be a bounded open set, u, v G C°(TL) satisfy 

u, v > 0, in Q, (44) 

and 

u > v on dtt. (45) 

We assume that there exist {A}, {A} C C°(Q,S nxn ) and positive functions {ui},{vi} C 
C 2 (fl) such that, for any compact subset K offl, 

A Ui + Pi G U, A Vi + t eU, in n, (46) 

Ui -> u, Vi^ v, (3i -> 0, A -> 0, in C°(K), (47) 

and 

F (A Vi + A) -> m C°(X). (48) 

TTien 

n > n on H. (49) 



To prove Proposition 3.1, we need to produce appropriate approximations to the {u{\. 
This is achieved by studying "the first variation" of the operator A u . 
Writing 

2 

W = U n- 2 , 

we have 

A" = A w := «;V 2 u> - i|Vw| 2 /. 
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Lemma 3.7 Let Q be a bounded open set in R n , w G C 2 (f2) satisfy, for some constant 
ci > ; 

w > ci, m Q, 

and ie£ 

V9(n) = e %|2 . (50) 

T/ien i/iere exists some constant 5 > 0, depending only on sup{|y| | y G fl}, and there 
exists e, depending only on 5, c\ and sup{|y| | y G Q}, such that for any < e < e, 

A w+e(p > (l + e^j A w + ^ipwl in Q. (51) 



Proof of Lemma 3.7. Let <p be a fixed function, a computation gives 
A w+eip = A w + e {wW 2 ip + y?V 2 w - Vw ■ Vc/?/} + e 2 A^. 

Replacing V 2 u> by w^ 1 ^A w + i|Vw| 2 /j in the above, we have 

A w+ev = (1 + e^)A w + e \wV 2 ip + ^^-tpl - Vw ■ Vipl) + e 2 A (p . (52) 
w [ 2w J 

For the ip in (50), 

Vip(y) = 26tp(y)y, V 2 cp(y) = 26tp(y)I + A5 2 (p(y)y <g> y. 

It follows that 

12 



Vw\ 2 1 T _ f „ iViol 2 /iViol 



wV 2 ip H (/?/ — Vtti • V<z»J 
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> tp{25w + -A8 2 w\y\ 2 \l. 
Aw 



It is clear that there exists 5 > 0, depending only on sup{|y| | y G Q}, such that 

„ 2 |Vw| 2 r „ „ r ^ . r |Vw| 2 r 

wv H — Vw • V^i > oynui H </JJ. 

2k; 4w 
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For this 5, there exists e > 0, depending only on 5, c\ and sup{|?/| | y G such that for 
all < e < e, 

A w+£(p > (l + e^j A w + edtpwl + e 2 A^ > (l + e^j A w + ^-tpwl, 



i.e. 

\ W 

Lemma 3.7 is established. 



□ 



Proof of Proposition 3.1. Since shrinking f2 slightly will not affect (45), we may assume 
without loss of generality that (47) and (48) hold with K replaced by Vt — from now on 
these equations will be understood in this sense. 

We prove (49) by contradiction argument. Suppose the contrary, then there exists 
some x G Q such that 

u(x) < v(x). 

It is clear that there exist < a < 1 and y G f2 such that 

u > av in Q, (53) 
u(y) = av(y). (54) 

Let 5 and ip be as in Lemma 3.7, 

Si :=su Pv /||A(|/)|| ^0, 

yen 



and 



By (51), 



Ui := [ ^ n 2 + €iip 



n-2 
2 



A Ui > ( 1 + a^-) A Ui + —ipwj, (55) 



Wi 



2 

" n-2 



where Wi = u { 

It is easy to see, using (53), (54), (45), and the convergence of Ui to u and Vi to v 

c (9l ^£ 
l ij fc V2' 2 



that for some 5 > and some large integer /, there exist, for i, j > I, G (§, ^f) and 



Dij G {y G f2 | dist(y, dil) > 5} such that 



Ui > ciijVj in 
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It follows that 

V (a^Ut) ( yij ) = Vvjiyij), (V 2 ^ 1 ^)^)) > ((V 2 ^)) , 

and 

A a ^( yij )<A^( yij ). 
Thus, in view of (55) and the definition of q, for some I > I, and for all i,j > I, 

A^( Vij ) + Pj( yij ) > {a tj )^A u ^y t3 )+~h(y tj ) 

> (dij)^ (l + 6i—) A Ui + (di^^-cpWil + Pjiyij) 

Fixing i = I, we have, for large j, 

^{ yij ) + Mvii) > K)^ (l + ^) ( AUi + ft) (Va) + (a/,)^^ W /. 
By (46), (2), (3), (6) and (7), 

/•'(•I" (///,) • ',(.</,,)) 

> ^(k-)^(i + ^)(^ + /?/)(^)) 

> min min F I b 1 1 + e? — ) (A u ' + /3 f ) («) ] > 
ven,dMt(y,an)>i ( „ )S 4 !I < b < ( ^ )S 4 !I \ V ™J/ / 

Sending j to oo leads to, in view of (48), that 

0=limF(A v i(y ij )+P j (y ij ))>0. 

Impossible. Proposition 3.1 is established. 

Remark 3.1 If we further assume in Proposition 3.1 that 

sup sup (|V 2 Ui| + |V 2 Vj|) < oo, 

i K V ' 

then modification of the proof of Proposition 3.2 yields a somewhat different proof. One 
observation is needed: In addition to (61), we have D 2 Ui )E > D 2 Vi )E on Si >e . So, for small 
s, using (2) and the continuity of D 2 u ite and D 2 v ite , we still have (62). 



□ 
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Proof of Theorem 1.2. We first prove that u G A. We only need to verify property 
(A2) since property (Al) has already been assumed. Let Q be as in the statement of (A2), 
then, by Lemma 3.5 and Lemma 3.6, both u x> \ and (1 + S)u are weak solutions of (8). 
Thus, by Proposition 3.1, (A2) is satisfied. So we have proved that u G A. By Theorem 
1.1, (12) holds for some a > 0,b > and x G R n . We only need to prove that 6 = 0. 
Suppose that b > 0, then a computation gives, for some positive constant (3, 

A u = (31 in R n . (56) 

Since u is a weak solution of (8), the hypotheses of Lemma 3.1 with h = are satisfied, 
and therefore, according to Lemma 3.1, 

A u G U in R n . (57) 

By (56), (57) and (4), 

A u = (31 G U in R n . 

Thus, by (7), 

F{A U ) > in R n , 
On the other hand, since A u G U, we apply Lemma 3.4 to obtain 

F(A U ) < 0. 

Impossible. We have proved that b = in (12) and therefore u = u(0). Theorem 1.2 is 
established. 

□ 

To prove Theorem 1.3 we need the following comparison principle for C 1 ' 1 solutions. 

Proposition 3.2 Let U C S nxn be a convex open set satisfying (2) and (3), F G C 1 (C/)n 
C°(U) satisfy (6) and (7), and let Q C R n be a bounded open set, u,v G C^l(Q) satisfy 
(44), (45) and 

F (A u ) > 0, F (A v ) = 0, almost everywhere in Q. (58) 

Then (49) holds. 

Proof. We prove it by contradition argument. We assume that min(w — v) < 0. Let (p 
be as in (50) for some fixed small S > 0, and let 

2_ n-2 

w := u ™- 2 , to e := it; + e<^, w e := (iu e J 2 . 
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Using Lemma 3.7 and (45), we can find some fixed small positive constants e and e\ such 
that 

A Uc > (l + e^j A u + SeJ, (59) 

and 

u e > v on dVL. 
Since A u e dU a.e. in f2, we have, using (2) and the openness of U, 

A Ue + M e U a.e. in Q, V M e <S nxn , ||M|| < 2ei. (60) 

By the contradiction hypothesis, u < v somewhere in fi, so there exists a e G (0, 1] such 
that 

u e > v e := a e v in Q, 
u e > v e on dQ., 

and 

5 £ := {x e n | « e (x) = v e (x)} ^ 0. 

Clealy, 

u e = v e , V« e = Vf e , on S'e. (61) 
Recall that e has been fixed. Let 

O s := {x e Q \ dist(x, S € ) < s}. 

By (61), 

A tu € + (l-t)Ve = tA Ue + (l_ t -) A V t Qn ^ 

So, in view the convexity and the openness of U, there exists a function i(s), ?(s) — > + 
as s — > + , such that 

A t Ue +(i-t)„ e e ^ a e in y ^ < t < L ^ 

Note that we have used, in deriving (62), the fact that A u is linear in V 2 u and both u 
and Vm are continuous. 

By (60), there exists some e 2 > 0, independent of s, such that 

dist (A tu ^ l ~ t)v %dU) > e 2 , V 1 - e 2 < t < 1. 

Thus, by (6), there exists some e 3 > 0, independent of s, such that 

F (A*^^ 1 -^') > e 3) > e 3 /, V 1 - e 2 < t < 1. (63) 
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Using the mean value theorem, in view of (63), we have 
e 3 - F ^4*0>K+(i-t(*)K^ 
< F (A Uf -) — F ^4*( s K+(i-f(sH^ = C \—F ^ tu e+(i-*H^ I ^ 

= : - ^ ^ aij(x,t)d^ 9ij(w £ - f e ) + bi(x)di(u e - v e ) + c(x)(u e - v e ), 
where aij(-,t),bi,c are bounded in L°° norm, and, in view of (6) and (63), 



(64) 



J_ dij(x, t)dtj > e A I 



for some e 4 > independent of s. In view of (64), we can find some small s > such that 
< ^e 3 < -ciijdijiuz - v e ) + bi(x)di(u e - v t ) + c(x)(u e - v e ), a.e. in O s , 

where a^, bi, c are in L°°(Os) and (a^) > e 4 J a.e. in O s . We know that 

u e — v e = on S € C O s , -u e — i; e > in O s , u t — v e > near dO s . 

But this violates the local maximum principle, see theorem 9.22 in [8] or theorem 4.8 in 
[1]. Proposition 3.2 is established. 

□ 

Proof of Theorem 1.3. We first prove that u G A. We only need to verify property 
(A2) since property (Al) has already been assumed. Let £1 be as in the statement of (A2), 
then, by the conformal invariance of A u , A u *^ and A 1 - 1 ^ are still in dU a.e. in f2. Thus, 
by Proposition 3.2, (A2) is satisfied. So we have proved that u e A. By Theorem 1.1, 
(12) holds for some a > 0, b > and x G R n . We only need to prove that b = 0. Suppose 
that b > 0, then A u is a positive constant multiple of / in R n , and therefore A u G U in 
R n according to (4). This violates A u G dU a.e. in R n . Theorem 1.3 is established. 

□ 



4 Proof of Theorem 1.4 and Theorem 1.5 

First we give a variation of Proposition 3.1 which allows u to have isolated singularities. 
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Proposition 4.1 Let U C S nxn be an open set satisfying (2) and (3), F G C l {U)nC°(U) 
satisfy (6) and (7), and let Q C R n be a bounded open set containing m points S m := 
{Pi, • • • , P m }, m>l,ue C°(U\S m ) and v e C^H) satisfy 

u > 0, Au < inQ\S m , v > in H, (65) 

u > v on dfl. (66) 
We assume that u is a weak solution of 

F(A U ) > mn\S m , 

v is a weak solution of 

F(A V ) = in Q \ S m . 

Then 

inf (u-v) > 0. (67) 



Proof. We prove it by induction on the number of points m. We start from m = with 
So = 0. The result is contained in Proposition 3.1. Now we assume that the result holds 
for m — 1 points, m — 1 > 0, and we will prove it for m points. 
Let 

Sm — {Pi i " - - j Pm} C fi. 

By the assumption on u and v, there exist {Pi}, {/?«} C C°(Q \ S m ,S nxn ) and positive 
functions {«;}, {vi} C C 2 (f2 \ S m ) such that, for any compact subset K of Q \ Sm, 

+foeU, A Vt + pieU, in ft \ S m , 

-> u, — > A -> 0, A -> 0, in C°(iT), 

and 

F(A Vi +Pi)^0 mC°(K). 
We prove (67) by contradiction argument. Suppose it does not hold, then 

inf (u — v) < 0. 

Since u > in Q \ S m , Au < in Q \ S m , we know that inf u > 0. Thus, for some 
< a < 1, 

inf (u — av) = 0. 
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Since we can use a 1 u instead of u, we may assume without loss of generality that a — 1. 
So we have, in addition, 

inf (u-v) = 0. (68) 

Let P m be the origin, and let 

Q:=n\{P 1 ,---,P m . 1 }. 

We fix some positive function <p so that the hypotheses on <p in Theorem 1.6, with Q being 
the fl there, are satisfied. Since $(f , 0, 1; •) — v and u > v on dQ, we can fix some small 
£4 > so that \x\ < 64 and |A — 1| < e 4 guarentee 

u> $(v,x,\;-) ondQ. (69) 

For such x and A, if we assume both 

inf [u-$(v,x,\;-)] =0 (70) 
n\{o} 



and 



liminf [u(y) - $(v,x,\;y)] > 0, 
Is/I — 



we would have, for some small e, e > 0, 

- $(v, x, A; y) > e > 0, V < |y| < e. (71) 

Let 

u(j/):=A 2 y?(A) u(y) : =A 2 u(x + Ay). 

We know from (69) and (71) that 

u>v ond{Q\Bi). 

It is easy to see that the hypotheses of Proposition 4.1, with u replaced by u, v replaced 
by v, Q replaced by Q \ B^, S m replaced by S m -\ := {Pi, • • • , P m _i}, are satisfied. By the 
induction hypothesis, 

_inf (u — v) > 0, 

(n\B e -)\5 m _i 

i.e. 

_inf [u - $(v,x, A; •)] > 0. 
(n\B- e )\s m -i 
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This and (71) violate (70). Impossible. Thus we have proved that (70) implies 

liminf [u(y) — &(v, x, A; y)\ =0. Namely, we have verified (24), with Q replaced by Q. 

Is/I — 

Therefore, by Theorem 1.6, either (25) holds or u — v — v(0) near the origin. 

If (25) holds, then ^inf (u — v) > for some e > 0. By the induction hypotheses, 

applied on Q \ B t , we obtain inf (u — v) > 0. It follows that inf (u — v) > 0, 

(n\B«)\{iv--,P m -i} n \ s ™ 
violating (68). A contradiction. 

If u — v — v(0) near the origin, say in Bi for some small e > 0, we let ip(y) = e 5 ^ be 
the function in Lemma 3.7, and let 

u e := (w ~ + eip) 2 , a e := inf — . 

n i/'i. -./',,} v 



XI 11 

Since — = — h 0(e), it is easy to see from (68) that 

v v 



lima e = l. (72) 



For \y\ < e, u(y) = v(y) = v(0). So 



^# = l-^ e ,(0)^e%l 2 +O( e 2 ). 
v(y) 2 

For e > small, 

— > max — — — > a e on B E / 2 . (73) 

V \y\=e v(y) 

Taking e > smaller if necessary, we have, in view of (72) and the fact u > v on dQ, 

— > a e on o\l. 

v 

Fix this e > and let Q := \ B ? / 2 . We know that 

(a e )~V > v on dn. (74) 

Let 



n-2 
2 



Making e > smaller if necessary, we have, by Lemma 3.7, u *EUinQ for large i. 

Clearly, (o e ) -1 ^ — > (a e )' 1 u <: in C z ° c (f2). By the induction hypothesis, in view of (74), we 
obtain 

inf ((a e )~V - f) > 0. 
n\{Pi,-,p m _i} 
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This and (73) imply 



inf 



u f 



> a f 



n\{Pi,-,p m } v 

violating the definition of a e . Impossible. Proposition 4.1 is established. 

Similar to Proposition 4.1, we have the following variation of Proposition 3.2. 



□ 



Proposition 4.2 Let U C S nxn be an open set satisfying (2) and (3), F G C\U)nC°(U) 
satisfy (6) and (7), and let Q C E™ fre a bounded open set containing m points S m := 
• • • , P m }, m > I, u e C°(n\ 5 m ) n C^(Q \ 5 m ) and u e C\9) n C^(Q \ 5 m ) safe/?/ 
ftfjj, ^ and 

F(A U ) > 0, = almost everywhere mVt\ S m . 

Then (67) holds. 

Proof. It follows from modification of the proof of Propsosition 4.1, using Proposition 3.2 
instead of Proposition 3.1, and making some other obvious changes. We omit the details. 

□ 

Now we give the 
Proof of Theorem 1.4. By (18) and the positivity of u, 

liminf-u(?/) > 0, liminf \y\ n ~ 2 u(y) > 0. 
Is/Ho \v\-hx) 

As usual, for any j;GI"\ {0} and for any < 5 < 1, 

X s (x) = sup{0 < /I < \x\ \u x>x (y) < (1 + 6)u(y),y < A < fi, \y - x\ > A, \y\ ^ 0} > 

is well defined. 

By the definition of \s(x), 



u. 



(75) 



Lemma 4.1 If \$(x) < \x\ for some < S < 1 and x G E n \ {0} ; then either 



lim inf 



[1 + 6)u(y) -u xMx) (y) 







or 



liminf \y\ n (1 + 5)u(y) - u xMx) {y) 
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Proof. Suppose the contrary, then for some < 5 < 5, x ER n \ {0}, X$(x) < \x\, and for 

3 

some R > 3\x\ + — , < e 2 < h min{|x|, \x\ — \$(x)}, e > 0, 

\x\ - X s (x) 

(1 + 8)u(y) - u x , x (y) > V |A - X s (x)\ < e 2 , \y - x\ > R, 

(1 + 5)u{y) - u X:X (y) > e, V |A - X s (x)\ <e 2 ,yeBi\ {0} U dB x . 

R 

Let 

n := {y E R n | - < \y\ < R, \y - x\ > X s (x)}, 
By Lemma 3.5 and Lemma 3.6, (1 + S)u is a weak solution of 



F =0 in n, 



and u x \( x ) is a weak solution of 

F(A U "- X «<*>) = infi. 

We also know that 

(1 + <J)u > M x ,A a(:c ) on D, and (1 + <f)u > u xMx) on dft. 

It follows, using Proposition 3.1, that 

(1 + S)u > u x - Xs(x) on n. 

As usual, the moving sphere procedure can go beyond Xg(x), violating the definition of 
Xs(x). Lemma 4.1 is established. 

□ 

Lemma 4.2 For all < 5 < 1 and for all x e M" \ {0}, A^(x) = 

Proof. We prove it by contradiction argument. Suppose the contrary, then Xs(x) < \x\ 
for some iGl"\{0}. Let 

n._ R _ P ._ T P - r 

y •- -dajWi .— x j— 7^ — , r 2 .— x, 

X 



u 



(y) := (1 + S)u xMx) (y), v(y) := 
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Since < \s(x) < \x\, we have Pi, P2 G f2. By (75), 

u>v in n\ {Pi, P 2 }. 

It is easy to check that the hypotheses of Proposition 4.1, with u replaced by u, v replaced 
by v, and with m — 2, are satisfied. Thus, by Proposition 4.1, 

inf (u — v) > 0. 

fA{Pi,P 2 } 



This implies 



lim inf 

Is/I — ^0 



1 + - w x ,Ai(x) (y) 



j; 



n-2 



lim inf — > 0, 



and 



liminf||/| n 2 [(1 + 5)u{y) - u xMx) (y)] = (\ 5 (x)) n 2 lim inf [u{z) - v(z)\ > 0, 



z^P 2 



which contradict to Lemma 4.1. Impossible. Lemma 4.2 is established. 

Now we complete the proof of Theorem 1.4. By Lemma 4.2, 

X 5 (x) = \x\, VieK"\{0},V0<5<l, 



□ 



i.e. 



UxM < ( x + s ) u (y), V < A < \x\, \y — x\ > X,y y^O. 
Sending 5 to leads to (19). The radial symmetry of u follows from (19), see e.g. [15]. 



Theorem 1.4 is established. 



□ 



Proof of Theorem 1.5. The proof is similar to that of Theorem 1.4, using Proposition 
3.2 instead of Proposition 3.1 and using Proposition 4.2 instead of Proposition 4.1. We 
omit the details. 



□ 



References 

[1] L. Caffarelli and X. Cabre, Fully nonlinear elliptic equations, American Mathematical 
Society Colloquium Publications 43, American Mathematical Society, Providence, 
RI, 1995. 



27 



[2] L. Caffarelli, B. Gidas and J. Spruck, Asymptotic symmetry and local behavior of 
semilinear elliptic equations with critical Sobolev growth, Comm. Pure Appl. Math. 
42 (1989), 271-297. 

[3] L. Caffarelli, L. Nirenberg and J. Spruck, The Dirichlet problem for nonlinear second- 
order elliptic equations, III: Functions of the eigenvalues of the Hessian. Acta Math. 
155 (1985), 261-301. 

[4] S.Y.A. Chang, Non-linear elliptic equations in conformal Geometry, Nachdiplom Lec- 
tures Course Notes, ETH, Zurich, Springer-Birkhauser, 2004. 

[5] S.Y.A. Chang, M. Gursky and P. Yang, An equation of Monge-Ampere type in 
conformal geometry, and four-manifolds of positive Ricci curvature, Ann. of Math. 
155 (2002), 709-787. 

[6] S.Y.A. Chang, M. Gursky and P. Yang, A prior estimate for a class of nonlinear 
equations on 4-manifolds, Journal D' Analyse Journal Mathematique 87 (2002), 151- 
186. 

[7] B. Gidas, W.M. Ni and L. Nirenberg, Symmetry and related properties via the max- 
imum principle, Comm. Math. Phys. 68 (1979), 209-243. 

[8] D. Gilbarg and N. Trudinger, Elliptic partial differential equations of second or- 
der. Second edition. Grundlehren der Mathematischen Wissenschaften [Fundamental 
Principles of Mathematical Sciences], 224. Springer- Verlag, Berlin- New York, 1983. 

[9] M. Gursky and J. Viaclovsky, Convexity and singularities of curvature equations in 
conformal geometry, arXiv:math.DG/0504066 vl 4 Apr 2005. 

[10] A. Li and Y.Y. Li, On some conformally invariant fully nonlinear equations, Comm. 
Pure Appl. Math. 56 (2003), 1416-1464. 

[11] A. Li, Liouville type theorem for some degenerate conformally invariant fully nonlin- 
ear equation, in preparation. 

[12] A. Li and Y.Y. Li, A general Liouville type theorem for some conformally invariant 
fully nonlinear equations, arXiv:math.AP/0301239 vl 21 Jan 2003. 

[13] A. Li and Y.Y. Li, Further results on Liouville type theorems for some conformally 
invariant fully nonlinear equations, arXiv:math.AP/0301254 vl 22 Jan 2003. 



28 



[14] A. Li and Y.Y. Li, On some conformally invariant fully nonlinear equations, Part II: 
Liouville, Harnack and Yamabe, arXiv:math.AP/0403442 vl 25 Mar 2004. 

[15] Y.Y. Li, Conformally invariant fully nonlinear elliptic equations and isolated singu- 
larities, preprint. 

[16] Y.Y. Li and L. Zhang, Liouville type theorems and Harnack type inequalities for 
semilinear elliptic equations, Journal d' Analyse Mathematique 90 (2003), 27-87. 

[17] J. Viaclovsky, Estimates and existence results for some fully nonlinear elliptic equa- 
tions on Riemannian manifolds, Comm. Anal. Geom. 10 (2002), 815-846. 



